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Preface 


I  hate  to  see  a  person  doing  extra  work  or  work  that  has  little  meaning  or  use.  I  chose  to 
discuss  Operational  Risk  Management  (ORM)  because  while  I  was  the  chief  of  flight  safety  at 
Cannon  AFB,  I  saw  people  developing  ORM  programs  that  where  a  waste  of  time.  I  was  one  of 
the  first  people  to  attend  the  Air  Combat  Command’s  (ACC)  ORM  training  at  Langley  AFB  and 
I  was  impressed  with  the  proactive  potential  of  ORM.  At  the  same  time,  I  was  skeptical  that 
ORM  would  become  a  misunderstood  program  rather  than  a  way  of  thinking.  During  this 
training,  the  commander  of  ACC  emphasized  that  ORM  would  be  a  philosophy  change  and  not  a 
new  program.  I  chose  this  topic  because  I  believe  people  are  creating  ORM  programs  without 
the  proper  understanding  of  the  limitations  of  these  ORM  programs  and  they  are  not  being 
integrated.  I  think  some  of  these  programs  are  forcing  people  to  do  more  work  for  little  gain. 

I  appreciate  the  help  that  I  received  on  this  project.  There  are  a  number  of  people  that  I 
talked  to  on  the  phone  to  gather  information  and  generally  discussed  ORM  issues  at  various 
ACC  bases.  Their  assistance  and  insight  was  invaluable  in  helping  me  develop  my  own  thoughts 
on  what  was  happening  throughout  ACC  with  respect  to  ORM,  and  to  ascertain  the  problems. 
Printed  information  on  risk  management  in  the  Air  University  library  was  basically  non-existent. 
Most  of  the  published  information  had  to  be  obtained  through  the  inter-library  loan  department, 
so  a  special  thanks  goes  out  to  Edith  Williams  for  her  help  in  getting  these  publications.  Finally, 
I’d  like  to  thank  my  research  advisor  for  his  help  and  multiple  suggestions  during  the  course  of 
this  paper.  With  his  help,  I  was  able  to  focus  my  thoughts  and  make  the  paper  easier  to  read. 
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Abstract 

In  1987,  the  US  Army  implemented  Operational  Risk  Management  (ORM)  teehniques  in  an 
attempt  to  reduce  mishaps.  By  the  early  1990s,  the  Army  reduced  serious  mishaps  by  more  than 
80  percent.  In  1996,  Air  Combat  Command  (ACC)  outlined  its  ORM  implementation  strategy  in 
an  attempt  to  reduce  all  types  of  mishaps  while  improving  operational  effectiveness.  While  this 
plan  allows  for  a  tailored  approach  to  a  wing’s  unique  culture,  the  implementation  has  created 
some  problems  within  ACC  units.  The  commander  of  ACC  emphasized  that  this  implementation 
is  a  top  down  change  in  philosophy  and  not  necessarily  a  new  safety  program.  This  is  important 
because  a  cultural  change  is  required  to  embed  ORM  seamlessly  into  everyday  missions.  The 
main  problem  found  during  this  study  was  that  units  are  attempting  to  quantify  risks  without  the 
necessary  information  or  knowledge  of  the  limitations  with  risk  quantification.  In  addition  to 
this  misguided  quantification,  units  are  not  properly  integrating  ORM  into  the  mission;  thus 
creating  extra  work  that  has  little  benefit.  ACC  is  taking  steps  to  improve  the  ORM  process  at 
the  unit  level  by  providing  better  information  databases  and  automated  risk  assessments  during 
mission  scheduling.  However,  these  programs  are  not  in  place  and  because  of  the  emphasis 
ORM  has  from  the  top  leadership,  ACC  units  are  basically  trying  to  do  too  much  too  soon.  This 
could  seriously  impede  the  ORM  implementation  plan  due  to  frustration  from  a  lack  of  progress 
with  regards  to  minimizing  risks  and  improving  mission  benefits.  This  study  recommends 
several  possible  solutions  to  these  problems,  but  emphasizes  that  more  study  is  needed  to  find  the 
best  possible  solutions  to  the  ORM  problems  in  ACC  units. 
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Chapter  1 


Introduction 


Operational  risk  management  is  a  logic-based,  common  sense  approach  to 
making  calculated  decisions  on  human,  material,  and  environmental  factors 
before,  during,  and  after  Air  Force  mission  activities  and  operations. 

— Air  Force  Instruction  91-213 
Operational  Risk  Management  (ORM)  Program 

The  goal  of  Operational  Risk  Management  (ORM)  is  to  enhance  mission  effectiveness  at  all 
levels,  while  preserving  assets  and  safeguarding  health  and  welfare.'  There  are  four  basic  ORM 
principles:  (1)  accept  no  unnecessary  risk,  (2)  make  risk  decisions  at  the  appropriate  level,  (3) 
accept  risks  when  benefits  outweigh  the  costs,  and  (4)  integrate  ORM  into  Air  Force  doctrine  and 
planning  at  all  levels.  To  accomplish  this  goal  using  these  principles,  the  Air  Combat 
Command  (ACC)  Commander  (COMACC)  has  challenged  everyone  in  ACC  to  use  the  six  step 
process  outlined  in  AFI  91-213  and  AFI  91-215  to  enhance  mission  effectiveness.  The  six  steps 
are:  (1)  Identify  Hazards,  (2)  Assess  the  Risk,  (3)  Analyze  Risk  Controls  Measures,  (4)  Make 
Control  Decisions,  (5)  Implement  Risk  Controls,  and  (6)  Supervise  and  Review  the  process. 
Since  1997,  squadrons  have  been  training  people  and  developing  techniques  for  implementing 
ORM  within  their  units.  The  COMACC  emphasized  that  this  implementation  was  a  change  in 
philosophy  and  not  necessarily  a  new  safety  program.  Top  leadership  touts  ORM  as  a  means  to 
further  reduce  accidents  and  preserve  our  scarce  resources  while  enhancing  mission 
effectiveness.  In  order  for  units  to  apply  the  four  principles  of  ORM,  they  need  to  develop  and 
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integrate  some  ORM  programs — ^this  ean  not  just  be  a  philosophy  ehange.  ACC  units  are 
developing  these  programs  and  they  are  ereating  problems  at  the  same  time.  If  the  problems 
continue,  they  could  impede  the  whole  ORM  implementation  process.  Starting  with  the  next 
chapter,  this  paper  will  address  some  of  the  ORM  programs  within  ACC  units.  Chapter  three 
will  detail  the  problems  inherent  with  these  programs,  and  finally  chapter  four  will  discuss  some 
possible  solutions  to  these  problems. 

Statement  of  the  Research  Question 

What  problems  exist  at  the  unit  level  in  ACC’s  implementation  of  ORM  and  how  can  they 
be  remedied? 


Background  and  Significance  of  the  Problem 

In  1987,  the  US  Army  implemented  Operational  Risk  Management  (ORM)  techniques  in  an 
attempt  to  reduce  mishaps.  By  the  early  1990s,  the  Army  reduced  serious  mishaps  by  more  than 
80  percent.  In  1996,  Air  Combat  Command  (ACC)  outlined  its  ORM  implementation  strategy  in 
an  attempt  to  reduce  all  types  of  mishaps  while  improving  operational  effectiveness.  While  this 
plan  allows  for  a  tailored  approach  to  a  wing’s  unique  culture,  the  implementation  has  created 
some  problems  within  ACC  units.  The  commander  of  ACC  emphasized  that  this  implementation 
is  a  top  down  change  in  philosophy  and  not  necessarily  a  new  safety  program.  This  is  important 
because  a  cultural  change  is  required  to  integrate  ORM  seamlessly  into  everyday  missions. 
However,  units  are  developing  ORM  programs  that  quantify  risks  and  these  programs  are 
creating  problems  because  they  do  not  understand  the  limitations  of  risk  quantification.  In 
addition  to  this  misguided  quantification,  units  are  not  properly  integrating  ORM  into  the 
mission;  thus  creating  extra  work  that  has  little  benefit.  ACC  is  taking  steps  to  improve  the 
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ORM  process  at  the  unit  level  by  providing  better  information  databases  and  automated  risk 
assessments  during  mission  seheduling.  However,  these  programs  are  not  in  plaee  and  beeause 
of  the  emphasis  ORM  has  from  the  top  leadership,  ACC  units  are  basically  trying  to  do  too  much 
too  soon.  This  could  seriously  impede  the  ORM  implementation  plan  due  to  frustration  from  a 
lack  of  progress  with  regards  to  minimizing  risks  and  improving  mission  benefits.  This  study 
reeommends  several  possible  solutions  to  these  problems  but  emphasizes  that  more  study  is 
needed  to  find  the  best  possible  solutions  to  the  ORM  problems  in  ACC. 

Definitions  and  Assumptions 

For  the  purpose  of  this  study,  definitions  from  the  Air  Foree  Instruction  on  ORM  will  be 
used.  Often  times  these  terms  will  vary  based  on  authors  and  context.  However,  most  are  fairly 
eonsistent  throughout  readings  on  risk  and  risk  management.  If  variations  exist,  it  will  be  noted 
in  this  text.  The  following  definitions  apply:  Hazard  is  any  real  or  potential  eondition  that  ean 
eause  mission  degradation,  injury,  illness,  or  death  to  personnel,  or  damage  to  or  loss  of 
equipment  or  property.  Risk  is  the  probability  and  severity  of  loss  from  exposure  to  the  hazard. 
Risk  is  sometimes  misused  to  represent  only  the  probability  of  an  event  oecurring.  Risk  must 
include  both  probability  and  eonsequenee.  In  quantitative  terms,  risk  is  the  produet  of  these  two 
elements.  Risk  assessment  is  the  applieation  of  quantitative  or  qualitative  measures  to 
determine  all  the  levels  of  risk  assoeiated  with  a  speeific  hazard  and  defines  the  probability, 
severity,  and  exposure  of  a  mishap  that  could  result  from  the  hazard."^ 

ASSUMPTIONS:  The  main  assumption  made  in  this  study  presumes  there  are  real  or 
pereeived  ORM  problems  at  the  unit  level  in  ACC.  The  purpose  of  this  researeh  was  to  identify 
some  of  the  problems  with  ORM,  not  the  level  to  which  the  problems  exist. 
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Notes 


^  Air  Force  Instruction  (AFI)  91-213,  Operational  Risk  Management  (ORM)  Program,  1 
September  1997,  2 

^  Air  Foree  Pamphlet  (AFP)  91-215,  Operational  Risk  Management  (ORM)  Guidelines  and 
Tools,  1  July  1998,  5-6 

^  Dr.  Conrow,  Edmund  H.,  The  Use  of  Ordinal  Scales  in  Defense  Systems  Engineering, 
“1995  Aequisition  Researeh  Symposium  Proeeedings,”  Defense  Systems  Management  College, 
June  1995,  2 

^  AFI  91-213,  2 
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Chapter  2 


Implementation  Efforts  to  Enhance  Operational  Risk  Management 

Philosophies 

Army,  Navy,  and  Marine  Implementation 

The  Army  took  the  lead  in  1987  with  its  implementation  of  ORM.  It  has  eontinued  to  be  a 
leader  in  enhaneing  ORM  philosophies.  Eaeh  of  the  other  services  adopted  very  similar  ORM 
implementation  plans  in  1996.  The  basic  implementation  concept  for  all  the  services  was 
service-wide  training  from  the  top  down,  integration  of  ORM  into  mission  critical  processes,  and 
finally  creating  a  cultural  change  in  the  way  missions  are  managed  to  weigh  the  risks  against  the 
benefits  and  make  informed  decisions  to  improve  combat  capabilities.  The  key  to  this 
implementation  is  training  from  the  top  down.  All  the  services  have  completed  most  of  their 
initial  training  throughout  their  respective  service  and  are  integrating  the  training  into  entry  level 
and  command  level  training  programs  to  perpetuate  ORM  philosophies.^  Each  of  the  services 
emphasize  in  training  that  ORM  is  not  a  new  safety  program,  rather  a  philosophy.  Basically  the 
services  are  looking  for  a  way  to  create  a  cultural  change.  This  is  a  sound  implementation  plan 
and  the  Air  Eorce  continues  with  its  efforts  to  implement  ORM. 
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Air  Force  ORM  Implementation 


The  Air  Force  implementation  plan  for  ORM  is  detailed  in  AFI  91-213  with  guidelines  and 
tools  described  in  AFP  91-215.  The  Air  Force  plan  is  a  top-down  approach.  The  ORM  steering 
committee  is  chaired  by  the  Air  Force  Chief  of  Safety  and  is  made  up  of  a  representative  from 
each  major  command  (MAJCOM)  (including  the  Air  National  Guard).  Higher  headquarters 
commanders  are  tasked  to  direct  the  development  of  their  command’s  ORM  programs  and 
appoint  ORM  managers.  The  Air  Force  ORM  training  program  begins  with  initial  military 
training  for  both  enlisted  and  officer  and  continues  at  the  unit  level.  Commanders  receive  ORM 
training  as  part  of  their  commander  training  course  and  supervisors  are  the  principle  advocates 
for  the  ORM  program  to  ensure  individuals  apply  ORM  on  a  day-to-day  basis."^  Clearly  the  Air 
Force  top-down  ORM  implementation  strategy  is  sound.  All  of  this  training  and  supervisory 
involvement  is  taking  place  in  ACC  units,  so  why  are  there  problems  with  the  ORM 
imp  lementation? 

It  may  not  be  a  problem  with  the  implementation  strategy  but  more  in  units  trying  to  do  too 
much  implementation  too  soon.  Although  AFI  91-213  does  not  set  a  timeline  for 
implementation,  an  implied  timeline  does  exist.  In  general  terms  the  first  step  is  training,  the 
second  step  is  integration,  and  the  third  step  is  the  desired  cultural  change  to  achieve  the  ORM 
goals.  The  training  details  ORM  expectations,  goals,  principles,  fundamentals  of  risk 
management,  and  tools  and  techniques  to  apply  what  is  learned.  The  ORM  process,  coupled 
with  the  tools  and  techniques,  are  the  items  needed  to  integrate  ORM  into  operations.  Once  there 
is  true  integration,  not  separate  ORM  processes,  then  the  units  are  well  on  their  way  to  cultural 
change.  True  integration  becomes  the  barrier  to  cultural  change.  Training  becomes  a  barrier  to 
integration.  The  Air  Force  has  made  exceptional  efforts  to  enhance  ORM  training.  The  Air 
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Force  needs  to  make  exeeptional  efforts  to  enhance  ORM  integration  before  they  can  expect 
units  to  achieve  an  ORM  eultural  ehange.  One  problem  this  study  found  is  that  units  are  not 
truly  integrating  ORM,  they  are  creating  additional  ORM  processes  at  the  wrong  point  in 
the  mission  process.  Units  are  creating  quantitative  measures  of  risks  without  the  proper 
baekground  information  to  do  so.  The  units  are  getting  ahead  of  the  implied  ORM 
implementation  timeline  and  that  eould  impede  the  whole  ORM  initiative  to  gain  a  cultural 
change.  So,  what  is  happening  within  ACC  units  that  may  impede  ORM  implementation  from  a 
perspective  of  a  top-down  strategy  on  training,  integration,  and  cultural  change? 

ACC  ORM  at  the  unit  level 

This  study  gathered  information  from  eaeh  ACC  wing  by  talking  to  the  safety  offices  and 
discussing  ORM  at  the  base.  Most  bases  have  an  individual  designated  as  the  focal  point  for 
ORM  issues  within  the  wing.  From  this  discussion  I  learned  that  most  wings  have  an  ORM 
program  designed  to  help  the  squadrons  work  through  ORM  issues  and  assist  in  the  integration 
of  ORM.  Some,  but  not  all,  of  the  wings  were  using  some  of  the  tools  and  resourees  outlined  in 
AFP  91-215.  The  bottom  line  is  ACC  units  are  adhering  to  the  top-down  implementation 
philosophy.  Wing  supervision  is  involved  with  ORM  and  advocates  the  use  of  ORM  techniques 
for  day-to-day  operations.  With  this  emphasis,  base  personnel  receive  ORM  training  from 
various  sources  and  attempt  to  implement  what  they  have  learned  in  their  respective  missions. 

Training:  Air  Foree  ORM  training  is  a  top-down  effort  starting  with  senior  leadership  and 
mandates  for  all  Air  Foree  personnel  to  have  received  ORM  training  by  1  Oetober  1998.  ACC 
has  done  its  part  in  maintaining  the  perpetual  nature  of  ORM  training.  ACC  wings  elaim  that 
nearly  everyone  has  received  at  least  the  Awareness  Training  as  outlined  AFI  91-213.  Eaeh 
wing  has  taken  the  extra  effort  to  either  send  people  to  advanced  ORM  training  or  have  ORM 
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teams  come  to  the  base  to  do  on-site  training.  Eighth  Air  Force  has  an  excellent  on-site  training 
program  and  team.^  Individuals  attending  these  weeklong  courses  are  usually  mid-level 
supervisors.  New  squadron  commanders  receive  basic  ORM  training  during  ACC's  Squadron 
Commanders  Course.^  As  a  whole,  ACC  is  doing  very  well  with  perpetuating  ORM 
philosophies  through  training. 

Integration:  ORM  integration  essentially  means  that  the  ORM  process  is  internal  to 
mission  procedures.  ACC  units  are  creating  ORM  processes  hoping  they  will  be  accepted  as  part 
of  the  mission  procedure.  However,  they  are  still  separate  processes,  which  generally  are  not 
being  accepted.  The  most  obvious  case  in  point  is  the  use  of  risk  assessment  matrices. 
Examples  are  shown  in  Appendix  A. 

Several  ACC  units  are  developing  and  using  ORM  worksheets  as  an  identification  and 
analysis  tool.  To  build  the  worksheets,  units  collect  information  from  current  guidance, 
experienced  unit  personnel,  and  a  variety  of  other  sources.  Matrices  are  a  collection  of  critical 
elements  within  the  mission  and  summed-up  to  assess  risk  and  elevate  decisions  to  an 
appropriate  level  to  establish  accountability.  The  matrices  are  completed  prior  to  the  mission,  so 
it  appears  to  be  a  proactive  process  to  mitigate  risks.  The  idea  seems  to  apply  all  the  principles 
of  ORM,  but  the  process  is  not  being  accepted  as  integral  to  the  mission.  Often  times  it  is 
thought  of  as  “just  another  piece  of  paper  to  fdl  out,”  or  individuals  ‘work’  the  numbers  so  the 
summarized  outcome  of  the  worksheet  reflects  their  own  intuitive  conclusion. 

ACC  units  are  using  three  different  levels  of  ORM:  (1)  Time-Critical,  (2)  Deliberate,  and  (3) 
Strategic.^  The  first  is  mental  or  intuitive  risk  management.  Deliberate  is  the  application  of  the 
6-step  ORM  process  (shown  in  Figure  1)  that  uses  experience  and  brainstorming  to  identify 
hazards  and  develop  controls  using  a  series  of  tools  and  techniques  as  detailed  in  AFP  91-215. 
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Strategic  ORM  uses  the  6-step  process  more  thoroughly  by  involving  research  and  analysis  from 
multiple  sources  such  as  databases  and  statistical  experience.  The  difference  between  the  three  is 
the  level  of  analysis,  manpower,  and  time  allocated  to  the  ORM  process. 


Figure  1.  Operational  Risk  Management  6-step  Process 

As  stated,  ACC  units  are  using  all  three  levels  of  ORM.  Time-critical  ORM  is  being  applied 
daily  to  operations,  but  then  again,  ACC  units  have  been  doing  this  intuitive  risk  analysis  for 
years  which  has  helped  to  bring  their  accident  rates  down  to  the  low  values  seen  today.  This 
study  found  several  examples  of  deliberate  ORM.  When  the  Eight  Air  Force  ORM  training  team 
goes  to  a  base,  they  have  the  class  of  about  50  people  perform  deliberate  ORM  on  an  important 
process  on  that  base.  This  type  of  training  shows  base  personnel  and  supervisors  they  can  set  up 
ORM  committees  to  run  through  the  ORM  process  and  produce  useful  information  for  the  base. 
These  committees  can  easily  produce  detailed  risk  analysis  for  specific  issues.  For  example, 
Barksdale  AFB  weapons  storage  was  scrutinized  using  some  of  the  ORM  tools  and  this 
committee  developed  a  detailed  analysis  to  improve  how  weapons  were  stored  inside  the 
building.  Decisions  were  made  based  on  this  committee’s  recommendations  and  changes  were 
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made  to  the  storage  proeedures.  They  later  learned  they  had  not  considered  all  the  risks  and  they 

g 

had  a  few  incidents.  However,  they  learned  from  their  mistakes  and  improved  on  the  analysis. 
The  bottom  line  is  this  committee  addressed  an  issue  that  was  considered  important  and  the  unit 
chose  to  spend  man-hours  to  find  reasonable  solutions.  This  is  a  form  of  ORM  integration;  a 
conscience  choice  to  improve  the  workplace  and  allocate  resources  to  make  it  happen.  If  this 
unit  had  chosen  to  make  each  weapons  individual  fill  out  a  worksheet  each  time  they  stored  a 
weapon,  they  would  not  have  been  integrating  ORM  but  adding  it  to  the  process  of  storing 
weapons.  ORM  committees  are  a  useful  tool  in  addressing  specific  issues  and  integrating 
deliberate  ORM  at  the  right  point  into  the  mission  process. 

Finally,  strategic  ORM  is  not  being  used  much  within  ACC  units  because  the  informational 
databases  do  not  exist  or  are  too  difficult  to  retrieve  and  analyze  data.  Cannon  AFB  performed  a 
strategic  ORM  process  on  its  Bird/ Aircraft  Strike  Hazard  (BASH)  program  by  combining  three 
separate  databases  into  one  analysis.^  However,  to  do  this  they  had  to  create  their  own  historical 
database  because  the  Air  Force  BASH  database  was  not  capable  of  numerical  analysis  and  data 
manipulation,  only  search  and  retrieve.  Cannon  AFB  was  able  to  integrate  strategic  BASH  ORM 
into  the  daily  mission  process  by  applying  risk  controls  ahead  of  the  mission  scheduling  process 
which  eliminated  the  need  for  aircrews  to  perform  a  strategic  BASH  analysis  on  a  daily  basis. 
However,  aircrews  still  used  intuitive  measures  to  assess  risks  during  daily  flight  operations, 
avoided  hazardous  environments,  and  channeled  BASH  information  to  the  supervisor  of  flying 
for  other  aircrews.  This  is  an  example  of  varying  levels  of  ORM  being  applied  at  different  points 
in  the  mission  process.  ACC  units  lack  the  necessary  information  databases  and  analysis  of 
those  databases  to  generate  effective  strategic  ORM.  With  the  definition  of  strategic  ORM  in 
AFI  91-213,  this  is  the  only  level  of  ORM  that  can  truly  be  considered  quantitative.  Both  time- 
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critical  and  deliberate  levels  are  intuitive  and  experiential  in  nature.  ACC  units  are  attempting  to 
apply  quantitative  analysis  to  issues  that  are  not  quantifiable,  and  that  will  be  discussed  in  the 
next  chapter. 

Cultural  Change:  Finally,  cultural  change  or  acceptance  of  ORM  philosophies  in  ACC 
units  is  not  overly  high.  This  study  did  not  assess  the  level  of  acceptance,  only  that  ORM  is  not 
fully  accepted  as  a  mainstream  process.  Although  not  statistically  proven  in  this  study,  people 
consider  ORM  a  safety  program,  not  a  new  philosophy.  However,  progress  toward  a  cultural 
change  is  taking  place  because  of  the  training  people  have  received  so  far.  Because  of  this 
training,  people  in  ACC  understand  that  ORM  exists  and  they  are  attempting  to  finds  ways  to 
apply  it  in  their  workplace.  An  ACC  ORM  cultural  change  has  a  long  way  to  go  before  it  will  be 
considered  integral  to  the  mission  rather  than  just  another  safety  program.  Continued  training 
will  help,  but  ORM  integration  is  the  next  key  barrier  to  this  cultural  change. 
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Chapter  3 


ORM  Problems  in  ACC  Units 

From  the  discussion  in  Chapter  two,  two  main  problems  were  identified  that  will  be 
addressed  in  this  chapter.  The  first  is  a  lack  of  integration  of  ORM  into  the  mainstream 
mission  process  and  second,  the  use  or  misuse  of  risk  quantification.  Integration  is  the  key 
barrier  to  cultural  change.  Additionally,  the  misuse  of  risk  quantification  is  indicative  of  the  lack 
of  or  misuse  of  risk  information  and  demonstrates  that  units  are  attempting  to  do  too  much  too 
soon  in  the  implementation  process. 

ORM  Cultural  Change  by  way  of  Integration 

A  key  objective  of  ORM  is  to  accomplish  the  ORM  process  as  an  integrated  aspect 
of  mainstream  mission  processes.  When  ORM  is  effectively  integrated,  it  quickly 
ceases  to  be  consciously  identifiable  as  a  separate  process. 

—Air  Force  Pamphlet  91-215 
Operational  Risk  Management  (ORM)  Guidelines  and  Tools 

Cultural  change  is  a  natural  byproduct  of  ORM  integration.  Process  integration  will  produce 
cultural  change.^  AFP  91-215  does  not  go  into  detail  how  units  should  integrate  ORM  other  than 
to  say  that  ORM  should  be  integrated  into  the  planning  process.  There  are  three  ORM 
integration  problems  discussed  in  this  chapter:  1)  the  ORM  process  versus  the  mission  process, 
2)  resource  allocation,  and  3)  cultural  acceptance  to  perform  the  tasks  required  to  integrate  ORM. 
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A  fourth  area  of  ORM  integration  is  training,  but  ACC  has  already  taken  eonsiderable  steps  to 
enhanee  and  perpetuate  ORM  training  so  it  is  not  discussed  as  a  problem. 

The  ORM  6-step  process  is  just  that,  a  process  which  takes  resources,  training,  and 
acceptance  to  perform.  That  process  must  be  done  somewhere  in  the  mission  process  and  yet  be 
integrated  into  the  daily  operations  of  the  mission  so  that  it  no  longer  is  seen  as  separate.  An 
additional  ORM  process  must  be  a  conscience  choice  by  the  mission  supervisor  to  allocate 
additional  resources  such  as  time,  people,  and  information  to  perform  the  ORM  function.  Hence 
resource  allocation  becomes  important  to  ORM  integration.  Training  is  required  to  perform  the 
ORM  process.  Finally,  the  concepts  of  ORM  must  be  culturally  accepted  within  the  organization 
for  it  to  be  seriously  considered  during  the  mission  process.  Each  of  these  areas  are  essential  to 
the  integration  of  ORM  into  the  mainstream  mission  process.  If  any  one  element  is  degraded,  the 
overall  affect  on  ORM  integration  is  degraded. 

Integrating  ORM  into  the  Mission  Process 

Integrating  the  ORM  process  into  the  mission  process  is  not  an  easy  matter.  What  does  it 
mean  to  integrate  ORM?  Where  in  the  mission  process  do  you  integrate  ORM?  For  starters, 
integrating  ORM  means  integrating  either  deliberate  or  strategic  levels  of  ORM  principles. 
Time-critical  ORM  is  not  something  that  is  normally  done  on  paper;  it  is  purely  an  intuitive, 
time-sensitive  level  of  analysis.  Time-critical  ORM  is  achieved  through  ORM  training  and 
acceptances  of  ORM  principles  so  people  will  perform  some  form  of  intuitive  ORM  in  time 
critical  situations.  When  ORM  is  integrated,  it  does  not  mean  the  ORM  6-step  process  has  to  be 
conducted  at  a  single  point  in  the  mission  process.  There  could  be  several  levels  of  ORM 
throughout  the  mission  process.  Figure  2  shows  a  simple  view  of  a  typical  mission  process. 
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Figure  2.  Typical  Mission  Process 

Generally,  each  phase  of  the  mission  process  passes  through  a  filter  (of  sorts)  which  applies 
regulations  and  rules  on  that  and  later  phases  of  the  mission.  When  the  mission  is  executed  it  has 
all  of  these  inherent  rules  to  keep  the  mission  from  failing.  Note  that  this  is  a  simple  view  and 
there  are  several  means  to  go  backwards  on  this  mission  flow  to  make  changes  if  required  to 
enhance  mission  effectiveness.  If  the  mission  is  debriefed  then  individuals  learn  lessons  so  the 
mission  can  be  improved  the  next  time  it  is  executed.  These  individual  lessons  apply  to  the 
mission  phases  and  not  necessarily  to  the  filters  that  impose  restrictions  on  the  mission  except  for 
possibly  at  the  execution  phase.  Most  of  the  learning  and  feedback  takes  place  within  the 
mission  planning  through  mission  debrief  phases.  ACC  units  do  not  have  a  consolidated  method 
for  feedback  throughout  the  entire  mission  process.  ORM  needs  that  feedback  method  to  be 
integrated  into  the  mission.  Recall  that  ORM  step  6  (Supervise  and  Review)  is  essentially  a 
feedback  loop  so  that  ORM  can  perpetuate  and  improve  as  the  cycle  iterates.  Figure  3  shows  a 
general  example  of  how  ORM  can  be  integrated  into  the  mission.  Different  levels  of  ORM  are 
applied  at  different  points  in  the  process  to  minimize  the  rules  and  replace  them  with  risk 
controls.  (Often  times  these  risk  controls  will  be  similar  to  the  rules  they  replace.  The  difference 
is  the  risk  controls  are  based  on  risk  and  benefit  analysis  rather  than  rules  not  based  on  this 
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analysis.)  In  this  example,  the  mission  is  exeeuted  with  a  list  of  risk  controls  imposed  on  the 
mission.  During  the  mission  debrief  these  risk  controls  are  evaluated  for  their  impact  on  mission 
success.  This  is  where  the  benefits  analysis  gains  attention.  Units  are  already  doing  some  of 
these  debrief  tasks  such  as  incident  reports,  in-flight  emergency  (IFE)  worksheets,  and 
airspace/range  utilization  sheets  to  name  a  few.  The  problem  is  that  none  of  this  data  is  being 
collected  and  consolidated  for  risk/benefit  analysis.  The  feedback  loop  does  not  emphasize  risk 
analysis  and  therefore  there  is  no  procedure  for  the  varying  levels  of  ORM  to  get  needed 
information  to  do  the  ORM  process  ahead  of  the  mission.  At  some  bases,  units  are  placing  a  risk 
worksheet  at  an  inappropriate  point  in  the  mission  process  and  the  foundation  (information  loop) 
of  the  ORM  function  within  the  mission  does  not  exist.  The  input  to  integrated  ORM  is  risk  and 
benefit  lessons  learned  databases  with  a  means  of  data  analysis.  The  output  of  integrated  ORM 
is  risk  controls  at  various  points  along  the  mission  process.  ORM  procedures  in  ACC  are  not 
populating  a  risk  database  or  creating  feedback  necessary  to  iterate  the  ORM  process  within  the 
mission  process.  Thus,  there  is  a  lack  of  ORM  integration  within  ACC. 

ORM  Resource  Allocation 

AFP  91-215  states:  "To  effectively  apply  risk  management,  commanders  must  dedicate  time 
and  resources  to  incorporate  risk  management  principles  into  the  planning  process."  ORM 
resource  allocations  involve  time,  information,  and  people.  AFP  91-215  describes  three  levels  of 
risk  management  that  can  be  applied  to  a  task:  time-critical,  deliberate,  and  strategic.  Therefore, 
the  level  of  ORM  determines  how  much  time  should  be  allocated  to  the  ORM  function.  The 
more  involved  the  ORM  process,  the  more  time,  information,  and  people  are  required  to  perform 
the  task. 
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Range  Regs  ROE,  Tactics 

API  1 1-F-XX  Regs  Airspace  Target  Flow 


-  Impact  of  Risk  Controls  and  mission  benefits  flows  back  up  the  line  to  enhance  future  ORM  assessments 

-  Risk  and  Benefit  Database  collection 


Figure  3,  ORM  Integrated  into  Mission  Process 

Information  is  critical  to  the  ORM  process.  There  are  multiple  sources  and  countless  tools 
and  techniques  to  manage  and  manipulate  information.  For  strategie  or  deliberate  risk 
management,  broad  mission  critical  risk  information  is  required  to  perform  analytical  risk 
assessments;  otherwise  the  assessment  falls  back  to  an  intuitive-experiential  based  assessment  by 
the  people  performing  the  analysis.  For  squadrons  to  bridge  the  gap  from  intuitive  to 
quantitative  risk  management,  they  first  need  integrated  information  systems.  These  systems  do 
not  yet  exist.  The  Air  Foree  is  working  with  the  Army  to  develop  a  Risk  Management 
Information  System  (RMIS).  The  Air  Force  has  several  databases  available  depending  on  the 
mission  and  the  type  of  information  required  to  perform  the  ORM  analysis.  The  most  obvious  is 
the  Air  Force  Safety  Center’s  accident  database.  This  database  ean  be  manipulated  to  obtain  and 
quantify  risk  factors  such  as  probability  and  consequence.  The  database  is  difficult  to  use  and 
the  RMIS  should  replace  it  with  more  manageable  risk  information.  There  are  hundreds  of 
databases  maintained  at  each  base  for  various  tasks  (i.e.  maintenanee,  parts,  erew  duties,  range 
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utilization,  bomb  tracking,  etc.).  These  databases  are  not  designed  for  quantitative  risk 
analysis — ^that  is,  the  identification  of  probability  and  consequence.  Therefore,  units  laek  the 
necessary  informational  databases  to  perform  quantitative  risk  analysis. 

Finally,  trained  people  are  a  required  resource  to  perform  the  ORM  process.  ACC  ORM 
training  is  designed  to  give  a  majority  of  the  people  a  basic  understanding  of  ORM  principles 
and  a  few  people  detailed  ORM  training.  ACC  squadrons  are  using  people  extensively  to  build 
and  develop  ORM  information.  One  eommon  alloeation  throughout  ACC  is  to  use  people  during 
ACC  direeted  safety  days.  These  safety  days  are  used  to  generate  functional  area  hazard  lists, 
which  are  prioritized  and  up-channeled  to  the  wing  for  possible  analysis.  While  these  lists  have 
generated  valuable  information,  having  the  entire  wing  generate  hazard  lists  every  safety  day  has 
rapidly  lost  its  eharm.  The  lists  are  repetitive  and  often  degenerate  down  to  the  level  of  nuisance 
hazards  sueh  as  traffic  and  parking  lot  issues.  Squadrons  could  make  mueh  better  use  of  this 
resource  in  developing  better  ways  to  integrate  ORM  rather  than  just  producing  repetitive  hazard 
lists.  ACC  does  not  have  enough  people  trained  to  the  level  of  detail  required  to  perform 
strategic  ORM  principles.  Basically,  people  are  trained  at  a  introductory  level  and  are  being 
asked  to  analyze  information  whieh  is  not  designed  to  be  used  for  risk  analysis  and  the  proeess  is 
not  being  integrated  whieh  is  creating  extra  ineffective  ORM  work. 

ORM  Cultural  Acceptance 

Cultural  acceptance  means  people  at  all  levels  are  willing  to  perform  the  ORM  tasks.  In  an 
artiele  for  the  Army's  FlightFax  magazine,  eultural  ehange  to  ORM  has  four  key  barriers:  (1) 
smaller  foree  with  more  missions,  (2)  personalities,  (3)  eompetition,  and  (4)  career  aspirations."^ 
The  Air  Force  barriers  can  easily  fit  into  these  eategories  as  well.  The  Air  Force  has  certainly 
been  a  part  of  the  "do  more  with  less"  military  draw  downs  since  the  beginning  of  the  1990's. 
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Changing  the  minds  of  mid-level  managers  from  the  mindset  of  "that's  the  way  we've  always 
done  it"  to  incorporating  ORM  into  the  process  creates  a  difficult  personality  barrier  to  cultural 
change.  Additionally  with  competition,  people  do  not  want  to  be  out  done  or  passed  over, 
therefore,  finding  a  way  to  make  ORM  integration  competitive  could  boost  cultural  acceptance. 
In  an  on-going  study  of  how  to  implement  safety  programs,  Dan  Peterson  says;  “what  gets 
measured  and  rewarded  gets  done.”^  Career  aspirations  also  can  affect  what  gets  done  in  the 
military.  People  will  go  to  great  lengths  to  accomplish  tasks  that  are  perceived  as  beneficial  to 
their  career.  One  case  in  point  is  to  acquire  a  masters  degree.  Military  officers  have  spent 
countless  hours  of  their  own  time  getting  a  degree  just  to  “fill  a  promotion  square.”  With  these 
four  barriers,  implementing  ORM  is  no  different. 

ORM  is  important  and  needs  to  gain  cultural  acceptance.  General  Ryan,  Air  Force  Chief  of 
Staff  said  in  his  1998  Air  Force  Posture  Statement  that  "ORM  is  key  to  maintaining  readiness  in 
peacetime,  dominance  in  combat,  and  a  crucial  component  for  force  protection.".^  In  AFI  91- 
213,  it  says  that  ORM  applies  to  all  Air  Force  personnel  and  functional  areas  including  the  Air 
Force  Reserve  and  Air  National  Guard.’  If  ORM  is  so  important  then  measure  it  and  make  it 
accountable.  Getting  this  type  of  buy-in  is  certainly  difficult.  During  an  ACC  Safety 
Conference  I  attended  in  October  1997,  the  director  of  ACC  ORM  asked  if  wing  ORM  programs 
should  be  inspected— the  answer  was  a  resounding  NO!  This  was  at  a  time  when  Quality  Air 
Force  Assessments  (QAFA)  were  still  fresh  on  everyone's  mind.  QAFAs  were  tied  to 
promotions  and  the  inspection  (measurement)  was  how  it  was  tied.  If  a  wing  got  excellent  or 
outstanding  ratings,  these  were  certainly  included  on  individual  performance  reports,  which  were 
used  for  promotion  boards.  The  problem  is  finding  a  way  to  observe  and  measure  ORM 
functions  so  that  individuals  can  be  rewarded  in  a  way  they  perceive  is  beneficial  and  not  just 
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required  to  get  a  certain  score  so  they  can  have  it  recorded  for  promotions.  If  people  perceive  a 
true  benefit,  then  they  will  make  an  effort  to  perform  the  ORM  tasks  that  will  break  down  the 
barriers  listed  here.  ORM  cultural  acceptance  in  ACC  is  low  and  therefore  a  problem  to  ORM 
integration. 

In  summary,  ACC  units  have  problems  with  ORM  integration.  They  are  not  applying 
appropriate  levels  of  ORM  at  the  right  point  in  the  mission  process.  Units  are  not  collecting, 
consolidating,  and  analyzing  risk  information.  There  is  no  feedback  loop  to  emphasize  risk 
analysis  and  therefore  there  is  no  process  for  the  varying  levels  of  ORM  to  get  needed 
information  to  do  the  ORM  process  ahead  of  the  mission.  Units  do  not  have  the  necessary 
information  required  to  perform  strategic  and  deliberate  ORM  or  they  are  attempting  to  do  so 
with  an  inadequate  or  inappropriate  risk  database.  For  the  most  part,  ORM  resources  are  not  yet 
available  for  ACC  units  to  effectively  use,  yet  they  are  still  trying  to  apply  strategic  ORM 
principles  with  risk  worksheets.  Additionally,  performing  the  ORM  function  is  not  culturally 
accepted  because  there  is  no  incentive,  other  than  top-down  pressure  for  compliance,  to  perform 
the  ORM  tasks.  The  bottom  line  is  that  units  are  trying  to  apply  risk  management  principles 
before  the  necessary  risk  management  infrastructure  is  established.  The  worst  example  of  this  is 
the  use  of  worksheets  to  satisfy  a  commander’s  need  to  show  that  ORM  is  being  accomplished. 
The  next  section  details  the  limitations  and  problems  with  risk  quantification  techniques  in  ACC. 

Quantification  of  Risk  and  Risk  Matrices 

True  risk  quantification  is  the  product  of  the  probability  of  occurrence  with  the  consequence 
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of  occurrence.  The  Air  Force  outlines  one  form  of  risk  quantification  using  what  it  calls  a  "Risk 
Assessment  Code"  (RAC)  as  shown  in  Table  1.  A  RAC  is  usually  codified  as  a  two  digit 
alphanumeric  such  as  "2C."  The  "2"  represents  the  severity  of  occurrence  and  the  "C"  represents 
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the  probability  of  occurrence.  An  identical  risk  assessment  matrix  is  given  as  a  means  to  assess 
risks  for  a  specific  task  in  AFP  91-215,  Attachment  3  "Risk  Assessment  Tools,  Details,  and 
Examples."  In  this  AFP  91-215  attachment,  it  lists  three  limitations  and  concerns  with  the  use  of 
the  matrix:  (1)  subjectivity  both  in  terms  of  interpretation  of  the  matrix  categories  and  the 
interpretation  of  the  hazard  being  assessed;  (2)  inconsistency,  assessments  will  vary  based  on 
subjectivity  from  one  rater  to  another;  and  (3)  lack  of  a  range  of  rankings.^  AFP  91-215  goes  on 
to  expand  the  matrix  to  allow  more  rankings  as  shown  in  Table  2  below.  If  you  took  all  the 
numbers  in  Table  2  and  lined  them  up  along  a  scale  from  1  to  20,  you  would  have  an  ordinal 
scale  of  risks  because  the  values  would  be  based  on  probability  and  severity.  That  is  not  what  is 
happening  in  ACC  squadrons. 


Table  1.  Risk  Assessment  Code 


PROBABILITY 

LIKELY 

PROBABLE 

POSSIBLE 

UNLIKELY 

A 

B 

c 

D 

H 

Perm  Disability 

I 

1 

1 

2 

3 

O 

Partial  Disability 

II 

1 

2 

3 

4 

Lost  Workday 

III 

2 

3 

4 

5 

First  Aid  Req’d 

IV 

3 

4 

5 

5 

Source:  AFI  91-301,  “Air  Force  Occupational  and  Environmental  Safety,  Fire  Protection, 
and  Health  (AFOSH)  Program"  ,  1  June  1996 


Table  2.  Modified  Risk  Matrix 


PROBABILITY 

FREQUENT 

LIKELY 

OCCASIONAL 

SELDOM 

UNLIKELY 

A 

B 

c 

D 

E 

H 

CATASTROPHIC 

I 

1 

2 

6 

8 

12 

O 

CRITICAL 

II 

3 

4 

7 

11 

15 

MODERATE 

III 

5 

9 

10 

14 

16 

NEGLIGIBLE 

IV 

13 

17 

18 

19 

20 

Risk  Levels 

Source:  AFI  91-215,  "Operational  Risk  Management  (ORM)  Guidelines  and  Tools”,  1  July 
1998. 
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Squadrons  are  using  "risk"  scales  or  "risk"  assessment  matrices  to  determine  levels  of  "risks" 

for  various  tasks.  (The  term  “risk”  is  used  in  this  manner  to  illustrate  that  the  word  risk  is  being 

used,  but  they  do  not  include  the  probability  and  consequence  elements  of  risk.)  During  this 

research  I  acquired  multiple  worksheets  from  various  sources,  and  while  not  all  the  worksheets 

were  from  ACC  units,  they  all  had  similar  characteristics.  The  scales  are  ordinal  in  nature.  That 

is,  they  rank  various  elements  from  high  to  low,  or,  for  example,  they  use  a  scale  from,  1  to  5  to 

represent  rank  orderings  of  the  elements.  There  are  several  characteristics  of  ordinal  scales  that 

must  be  considered  when  using  them  to  make  decisions.  In  the  paper  "The  Use  of  Ordinal  scales 

in  Defense  Systems  Engineering,"  Dr.  Edmund  Conrow  makes  a  very  detailed  case  and  lists 

several  caveats  for  using  ordinal  scales  in  risk  analysis.  Eirst,  ordinal  scale  values  represent  rank 

orderings.  Typically  the  values  on  these  scales  do  not  represent  probability  and  consequence  to 

the  category  they  are  assigned  and  therefore  are  not  risk  values.  Generally  speaking,  ordinal 

scales  represent  uncertainty.  Second,  ordinal  scales  are  not  absolute,  nor  do  they  represent 

cardinal  numbers."  (A  number  which  represents  a  value  rather  than  a  ranking.)  Third,  ordinal 

scales  generally  have  arbitrary  intervals  between  scale  levels.  Eor  example,  a  score  of  5  is 

greater  than  2  but  not  2.5  times  greater.  Eorth,  ordinal  scales  are  monotonic  and  positive.  A 

score  of  5  is  greater  than  4  which  is  greater  than  3,  and  so  on.  Einally,  while  a  scale  level 

assigned  to  an  element  is  correct,  the  value  is  not  meaningful  beyond  an  ordering  sense. 

Generally  speaking,  ordinal  scales  are  not  calibrated  and  thus,  it  is  not  possible  to  perform  any 

mathematical  operations  with  any  degree  of  confidence.  Dr  Conrow  goes  on  to  assert: 

Given  multiple  ordinal  scales,  the  analyst  or  program  manager  is  often  tempted  to 
roll-up  the  scores  mathematically  into  a  single  value.  I  have  examined  a  variety 
of  normalization  schemes  and  averaging  techniques— none  of  them  yield 
meaningful  results.  In  general  it  is  recommended  that  raw  ordinal  scores  be  used 
in  the  program's  risk  management... tracking  process.  When  a  single  score  is 
needed  per  category,  then  the  highest  of  the  subcategory  scores  should  be  used. 
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Although  this  represents  a  conservative  approach,  it  prevents  potentially 
noteworthy  issues  from  being  masked  and  overlooked/^  (Emphasis  added) 

The  essence  of  Dr.  Conrow's  discussion  is  that  risk  matrices  cannot  be  summed  to  give  a  "risk 

factor"  or  "risk  gauge"  to  be  used  for  some  other  purpose.  In  ACC,  the  other  purpose  is  to  show 

that  “risks”  are  being  elevated  to  a  higher  authority,  which  satisfies  a  need  to  document  and  show 

a  paper  trail  for  accountability.  These  worksheets  would  not  stand  up  to  scrutiny  if  a  true 

accountability  issue  were  in  question.  Furthermore,  if  categories  of  scores  are  used  (Dr.  Conrow 

recommends  using  ordinal  scales  in  risk  management)  and  some  general  assessment  must  be 

made  on  the  whole  of  the  categories,  then  use  the  highest  score  of  all  the  categories  rather  than 

the  sum  of  the  scores.  Dr.  Conrow's  analysis  makes  it  very  clear  that  the  use  of  ordinal  scales, 

such  as  those  being  used  in  ACC  squadrons,  for  anything  more  than  the  relative  comparison  of 

one  category  to  another  is  meaningless.  The  matrices  that  I  found  being  used  in  squadrons 

categorize  hazardous  elements  of  a  task,  which  would  be  construed  as  hazards  not  risks.  The 

most  that  can  be  derived  from  these  matrices  is  that  a  score  in  one  category  may  be  relatively 

higher  than  another  category  based  on  the  assessment  made  by  the  individual  using  the  matrix. 

This  information  could  then  be  used  to  help  make  a  decision  during  the  mission  process. 

However,  rolling-up  all  the  categories  into  a  single  score  to  make  a  decision  is  inappropriate. 

Some  of  the  matrices  are  quite  in-depth  and  serve  a  very  useful  purpose  to  categorize 
hazardous  elements  of  the  task.  This  can  be  useful  in  highlighting  to  a  mission  manager  which 
elements  are  critical  to  the  task  so  effective  decisions  can  be  made.  Two  examples  of  these 
matrices  are  included  in  Appendix  A.  The  first  example  "Aircraft  Towing  Matrix"  basically  uses 
ordinal  scales  to  represent  "risks"  within  each  category.  The  manager  is  asked  to  circle  the 
appropriate  block  and  proceed  to  the  next  category  until  each  is  done.  Each  of  the  four 
categories  asks  generic,  yet  valuable  questions  that  give  a  sense  of  criticality  for  the  section  when 
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it  comes  to  performing  the  task  of  towing  an  aircraft.  Finally  the  manager  is  asked  to  add  the 
scores  and  compare  the  results  to  a  "risk  gauge"  at  the  bottom  which  assigns  a  level  of 
responsibility.  This  example  is  divided  into  four  categories.  As  an  example,  apply  the  values  4, 
2,  3,  4  for  categories  A  through  D  respectfully.  The  most  that  ean  be  said  about  this  matrix  is 
that  "Personnel  Experience"  (category  A  score  =  4)  is  eonsidered  more  hazardous  to  this  task 
than  categories  B  and  C,  and  similar  to  eategory  D.  The  summed  total  gives  a  score  of  13,  which 
equates  to  a  "risk  gauge"  that  is  off  the  eharts  and  requires  Maintenance  Supervision  to  be 
accountable  for  the  task.  The  seenario  that  produeed  these  numbers  is  simple:  An  inexperienced 
tow  erew  working  at  the  end  of  their  shift  (9  hours)  with  good  equipment,  which  they  use 
frequently,  are  about  to  tow  an  aircraft  using  radios  in  a  noisy,  congested  area  to  get  the  aircraft 
into  a  hanger  for  maintenance  because  the  winds  have  picked  up  to  25  knots.  In  my  experience 
with  maintenance,  this  is  not  an  off-the-scale  risky  seenario.  This  is  conjecture  on  my  part,  but 
the  point  is  that  simple  everyday  seenarios  can  produce  seemingly  high-risk  ventures  when 
applied  to  inappropriately  used  "risk  matrices."  Notice  on  this  example  that  even  if  all  the 
categories  where  scored  as  1,  the  risk  gauge  would  almost  exceed  the  Tow  Supervisor’s  level  of 
responsibility,  meaning  that  at  this  squadron  nearly  every  towing  would  have  to  be  "signed  off 
by  a  higher  authority.  This  is  easily  fixed  as  shown  in  the  seeond  example  where  the  low  end  of 
the  scale  is  zero.  What  should  be  said  about  this  first  example  is  the  tow  supervisor  is  about  to 
embark  on  a  task  with  a  "risk  factor"  of  4  (the  highest  of  the  four  categories).  What  he  does  with 
this  factor  would  depend  on  squadron  policy.  Some  ideas  that  eome  to  mind  are  to  mention  it  to 
the  flight  line  supervisor  (Bear  or  Ram  5  in  this  example)  and  quickly  brief  the  tow  erew  of  the 
extra  hazards  associated  with  the  high  winds,  congested  area,  use  of  radios,  and  an  inexperienced 
crew.  The  essence  of  the  matrix  is  still  aecomplished,  that  is,  task  critical  questions  were  asked. 
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potential  hazards  identified,  and  a  level  of  responsibility  was  assessed — ^all  qualitatively,  not 
quantitatively.  If  you  remove  the  quantitative  elements  from  this  example,  the  matrix  would 
reduce  to  a  set  of  questions  and  possible  answers.  In  other  words,  an  Aircraft  Towing  Briefing 
Guide  for  the  tow  supervisor.  These  questions  could  easily  be  placed  into  the  flight  line  work 
cards  and  either  mentally  assessed  at  the  time  of  the  task  or  briefed  to  the  whole  crew  depending 
on  the  situation.  The  bottom  line  is  that  an  inappropriate  quantitative  time-critical  ORM  process 
is  replaced  with  an  intuitive  time-critical  ORM  procedure  that  is  easily  integrated  into  the  towing 
mission  process. 

One  more  point  with  this  first  example.  Are  the  four  categories  the  only  critical  sets  of 
hazards  associated  with  towing  an  aircraft?  Depending  on  the  situation,  I  would  answer  no.  This 
matrix  could  create  the  perception  that  these  are  the  only  areas  that  need  to  be  assessed  when  in 
fact  there  could  easily  be  more  critical  factors  involved.  A  set  of  questions  or  a  simple  briefing 
(a  memory  jogger  to  the  critical  elements  of  the  task)  may  generate  more  questions  and  better 
assessments  based  on  the  situation  of  the  task  rather  than  a  pre-defined  matrix,  extra  paper  work, 
and  supervisory  signatures  required  before  a  task  can  even  begin.  Integration  of  critical  task 
assessments  does  not  come  from  extra  paper  work  or  signatures. 

The  next  example  in  Appendix  A  is  similar  because  it  essentially  uses  ordinal  scales,  yet 
different  because  the  worksheet  has  weighted  categories  horizontally  and  vertically  in  an  attempt 
to  calibrate  the  relative  strength  of  one  category  to  another.  Again  Dr.  Conrow  addresses  the 
issue  of  calibration  explicitly: 

Techniques  exist  that  may  be  used  to  calibrate  ordinal  scales,  such  as  the 
Analytical  Hierarchy  Process  (AHP).  (The  calibration  of  scale  levels  within  a 
single  ordinal  scale  can  be  viewed  as  vertical  calibration.  Calibration  (Weighting) 
between  scales  can  be  viewed  as  horizontal  calibration.)  While  this  may  provide 
a  relative  calibration  within  a  scale  for  different  levels,  or  between  scales,  it  will 
not  absolutely  calibrate  any  scale  level  or  weighting  between  scales...  the 
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calibrated  scores  are  not  probabilities,  and  the  absolute  magnitude  of  any  scale 
level  generally  remains  unknown.  ...The  lowest  scale  level  does  not  represent 
zero  probability  or  uncertainty  in  almost  any  instance.  In  addition,  it  is  not 
advisable  to  perform  mathematical  operations  on  scales  with  relative  calibration 
because  the  meaning  of  the  results  may  be  difficult  to  interpret.  ...it  is  generally 
of  greater  interest  to  note  each  score,  than  the  sum,  average,  or  other  combination 
of  the  scores 

First  Dr.  Conrow  defines  ordinal  scale  calibration  or  weighting  and  emphasizes  that  the 
calibrations  are  not  absolute  values,  nor  are  they  values  of  probabilities  that  can  be  combined 
with  a  consequence  element  to  obtain  risk  He  again  noted  these  numbers  should  not  be  summed 
nor  averaged  to  collect  a  single  representative  number  from  the  list  of  scores.  The  thing  Dr. 
Conrow  warns  against  is  precisely  what  is  taking  place  in  ACC  squadrons.  In  the  second 
example  in  Appendix  A,  each  category  scale  is  given  "points"  such  as  0,  20,  and  30  rather  than 
an  ordinal  scale  of  1,  2,  and  3.  Furthermore,  the  relative  weight  from  one  category  to  the  next 
varies.  The  "Excessive  Delay..."  category  has  a  scale  from  0  to  10,  whereas  the  "Mission 
Complexity"  category  has  a  scale  from  0  to  50.  This  weighting  within  and  between  scales  gives 
a  perception  of  calibration.  It  is  important  to  note,  this  is  only  a  perception  and  not  statistically 
sound  numerical  values.  All  the  matrices  that  were  weighted  were  derived  from  experience  and 
intuition  and  not  statistical  probabilities  that  one  category  carries  more  weight  than  another.  I 
would  offer  a  guess  to  the  uncertainty  of  the  weighting  to  be  on  the  order  of  ±  25  points,  or  half 
the  total  range  of  this  scale.  Again,  this  is  conjecture,  but  the  point  is  that  uncertainty  within 
these  matrices  is  relatively  high.  If  an  attempt  is  made  to  sum  these  figures  to  produce  a  "Level 
of  Approval  Required"  (as  in  this  example),  than  the  uncertainty  of  that  number  (which  also  must 
be  summed)  is  exceedingly  high  to  the  point  that  having  a  "Level  of  Approval"  scale  becomes 
meaningless.  This  example  is  indicative  of  others  that  I  have  seen  for  aircrew  and  flight 
operations. 
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Switching  to  the  positive  aspeets  of  matriees,  I  have  already  mentioned  a  couple  points. 
First,  matrices  ean  produee  a  list  of  critieal  elements  to  tasks  and  missions.  One  squadron 
produced  a  questionnaire  and  then  consolidated  the  answers  to  build  a  detailed  "risk"  matrix  to 
be  filled  out  during  eaeh  mission.  This  means  the  matrix  was  created  from  a  vast  eollection  of 
the  experience  within  the  wing.  Seeond,  matriees  can  show  a  rank  ordering  relationship  among 
the  categories  within  the  matrix,  and,  if  the  matrices  are  not  summed,  this  rank  ordering  can 
show  where  the  uncertainty  is  within  the  mission. 

Above,  I  alluded  to  the  faet  that  ordinal  scales  could  be  derived  from  true  risk  (ineorporating 
both  consequence  and  probability)  by  using  the  values  in  table  2  as  the  ordinal  scale.  If  these 
values  are  then  used,  it  may  be  possible  to  calibrate  the  matriees  using  the  Analytieal  Hierarehy 
Proeess  (AHP)  or  other  calibration  methods  to  provide  a  helpful  relative  calibration  within  a 
scale  for  different  levels,  or  between  different  ordinal  seales.^^  (This  study  does  not  address  how 
this  is  done.)  The  draw  back  is  a  very  task  intensive  and  information  dependent  analysis.  The 
need  for  detailed,  aceessible  databases  and  database  manipulation  is  required  to  make  this 
endeavor  work.  It  is  task  intensive  beeause  every  hazard  for  a  given  task  must  be  analyzed 
against  a  probability  and  eonsequence  matrix  as  shown  in  Table  2.  The  undertaking  would  be 
very  information  dependent  beeause  eaeh  hazard  would  need  historical  data  or  collaborated 
intuitive  data  in  order  to  perform  a  probability  and  consequence  analysis.  All  of  this  data  would 
need  to  be  stored  and  manipulated  using  a  powerful,  accessible  database  for  the  analysis.  The 
endeavor  to  calibrate  a  true  risk  matrix  is  enormous.  Yet  it  has  been  done.  In  the  article  Some 
Considerations  for  Implementing  Risk  Management  in  Defense  Programs,  Dr.  Conrow  and  Mark 
Fredrickson  wrote: 
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Risk  assessment  summary  level  eharts,  detailed  matrix  eharts,  and  quad  charts 
were  successfully  used  by  the  SPO  [System  Program  Office]  director  on  several 
occasions  to  brief  the  program  to  higher-level  DoD  and  Service  management. 

These  matrices  and  risk  assessment  summaries  were  first  modified  to  reduce  inconsistencies  and 

improve  assessment  accuracy.  The  SPO  in  this  case  factored  in  the  limitations  of  quantitative 

risk  analysis  and  the  limitations  of  matrices  to  produce  a  risk  management  process  that  was 

improved  over  a  3 -year  period.^'^ 

As  it  stands  now  in  ACC  squadrons,  the  matrices  that  have  been  developed  are  examples  of 
how  not  to  use  matrices  for  operational  risk  management  assessments.  They  are  ordinal  scales 
that  generally  yield  uncertainty  rather  than  risks.  The  numbers  used  within  each  category  are  not 
based  on  statistical  analysis  but  experiential  data,  which  can  be  misleading  and  misinterpreted 
when  filled  out  by  various  people.  The  matrices  attempt  to  sum  the  categories  to  determine 
which  level  of  supervision  is  required  for  the  task.  The  category  descriptions  and  scale 
descriptions  are  often  open  to  interpretation,  which  leads  to  more  uncertainty.  Finally,  these 
worksheets  undermine  the  attempt  to  integrate  ORM  into  the  squadron  by  creating  extra  work 
that  is  outside  the  mainstream  of  the  mission  process.  The  only  good  thing  these  worksheets 
have  in  common  is  they  identify  some,  but  not  all,  of  the  critical  elements  for  the  mission  they 
represent  and  a  means  for  inexperienced  people  to  reflect  on  the  thoughts  and  ideas  of  the 
experienced  people  who  put  the  matrices  together. 

Notes 

^  Dierberger,  Paul  A.,  The  Operational  Risk  Management  Course.  Transportation  Safety 
Institute,  OK.  February  1997.  Course  presented  to  Air  Combat  Command,  Langely  AFB,  VA, 
10-13  Feb  1997,  Page  19,  Slide  38  This  is  the  original  course  material  presented  to  ACC  initial 
cadre  of  ORM  champions  in  early  1997.  This  material  serves  as  the  basis  for  describing  ORM  as 
it  is  being  applied  and  taught  to  ACC  units. 

^  Air  Force  Pamphlet  (AFP)  91-215,  Operational  Risk  Management  (ORM)  Guidelines  and 
Tools,  1  July  1998,  13 
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John  D.  Phillips,  Headquarters  Air  Foree  Safety  Center,  Chief,  Operational  Risk 
Management,  Kirtland  AFB,  interviewed  by  author,  20  November  1998.  The  Army's  Risk 
Management  Information  System  ean  be  aecessed  (with  permission  by  justifying  information 
requirement)  at  http://rmis.armv.mil/ 

^  Konitzer,  Brig  Gen  Thomas  J.  The  Challenges  of  Change.  FlightFax  25  no.  1  (Oetober 
1996):  pages  2-4.  Addresses  ehange  in  a  smaller  Army  with  different  mission,  inereased  ops- 
tempo  and  leaner  budgets  by  using  ORM.  Talks  about  the  Army's  safety  cultural  dilemma  and 
asks  the  question  "Can  we  change  our  culture?"  Also  addresses  the  Army's  barriers  to  cultural 
change  with  respect  to  ORM. 

^  Peterson,  Dan,  The  Dan  Peterson  Safety  Management  Series,  Core  Media  Training 
Solutions,  1990,  chapter  3 

^  Air  Force  Posture  Statement  1998,  On-line,  8  Aug  1998,  Available  at 
http://www.af  mil/lib/afissues/1998/posture/page2.html 

^  Air  Force  Instruction  (AFI)  91-213,  Operational  Risk  Management  (ORM)  Program,  I 
September  1997,  I 
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Chapter  4 


Possible  Solutions  to  the  ORM  Problems  in  ACC  Units 

Chapter  three  diseussed  two  main  ORM  problems  in  ACC  units.  The  first  was  ORM 
integration  to  aehieve  ORM  eultural  ehange.  This  was  further  broken  down  into  three 
subeategories  of  proeess  integration,  resource  allocation,  and  acceptance.  The  second  main 
problem  was  the  misuse  of  risk  matrices  or  trying  to  quantify  risks  that  are  not  quantifiable. 
There  are  no  easy  solutions  to  these  problems.  This  chapter  briefly  discusses  some  possible 
solutions  (typed  in  bold)  to  help  in  the  implementation  of  ORM.  However,  the  most  likely 
solution  to  the  problems  will  only  come  from  further  study  and  analysis  of  ORM  at  the  unit  level. 
ORM  is  important  to  the  wellbeing  of  the  Air  Force  and  time  spent  studying  it  and  finding 
solutions  is  worth  the  effort. 


Integration  of  ORM 

Chapter  three  detailed  some  of  the  problems  with  ORM  integration  in  ACC  units  divided 
into  three  subcategories.  First,  units  were  not  integrating  the  ORM  process  at  appropriate  points 
in  the  mission  process  and  there  were  ineffective  feedback  methods  throughout  the  mission 
process  to  improve  risk  analysis.  The  input  to  integrated  ORM  is  information  and  its  analysis. 
The  output  of  integrated  ORM  is  risk  controls  at  various  points  along  the  mission  process.  The 
solution  to  this  problem  is  first  to  establish  where  in  the  mission  process  ORM  techniques  need 
to  be  applied  and  then  setup  a  method  to  collect,  manipulate,  and  analyze  information  about  the 
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mission  that  can  be  used  to  perpetuate  the  ORM  proeess.  In  finding  where  ORM  needs  to  be 

applied  to  the  mission  proeess,  API  91-213  states: 

Individuals  writing  doetrine  and  planning  should  apply  the  ORM  eoneepts.  The 
initial  development  and  annual  review  of  plans,  directives  and  other  guidanee 
provide  the  opportune  time  to  ensure  the  applieation  of  ORM  prineiples/ 

Other  than  this,  API  91-213  does  not  mention  anything  more  about  ORM  integration.  APP  91- 

215  discusses  ORM  integration  throughout  the  pamphlet.  Integration  is  the  key  to  cultural 

change;  more  details  need  to  he  taught  during  advanced  ORM  training.  These  instruetions 

are  eorrect  in  stating  that  ORM  needs  to  be  done  in  the  planning  phase,  but  they  need  to  expand 

what  level  of  ORM  at  whieh  level  of  planning  and  a  discussion  of  where  the  information  should 

come  from.  Pigure  3  (in  Chapter  three)  showed  a  possible  solution  for  this  type  of  ORM 

integration.  The  key  in  this  example  was  an  information  loop  after  the  mission  that  flowed  baek 

to  the  varying  levels  of  ORM  throughout  the  mission  proeess  to  apply  risk  eontrols  at  appropriate 

points  in  the  proeess.  Por  example,  strategie  ORM  is  used  to  determine  general  Go-No-Go 

eriteria  for  assigning  a  specific  mission  to  specific  individuals.  Once  this  is  done,  these  risk 

eontrols  apply  throughout  the  mission  proeess  and  do  not  have  to  be  done  again  during  the 

mission  planning  phase  or  mission  execution  phase.  ORM  needs  to  be  applied  at  appropriate 

points  in  the  mission  proeess.  The  information  feedbaek  must  be  developed  to  inelude  risk  and 

benefit  information  so  a  eomparison  can  be  made.  As  stated  earlier,  mueh  of  this  data  is  already 

being  eolleeted,  just  not  in  a  format  useful  to  risk  ealeulations.  Units  are  eolleeting  daily 

information  about  eaeh  mission  that  is  flown  to  include  hazards,  maintenanee,  range/air  spaee 

use,  and  so  on.  Rather  than  having  airerews  fill  out  a  risk  worksheet  prior  to  the  flight,  whieh 

has  little  impact,  have  them  fill  out  a  post  flight  analysis  sheet  to  be  eolleeted  for  future 

risk/benefit  assessments. 
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The  second  category  within  the  ORM  integration  problem  was  resource  allocation.  The 
main  problem  here  was  getting  information  worthy  of  risk  analysis.  For  the  most  part,  risk 
information  that  is  available  to  be  manipulated  and  analyzed  does  not  exist  in  ACC  units.  ACC 
Safety  and  the  Air  Force  Safety  offices  are  making  excellent  efforts  to  get  risk  information  down 
to  the  squadrons.  For  example,  ACC  is  already  testing  a  program  called  Squadron  Assistant/Risk 
Assessment  (SARA).  This  is  an  excellent  decision  aid  for  aircrew  scheduling  and  aircrew 
management  with  risk  management  built-in.  Basically  the  database  program  provides  automatic 
risk  calculations  on  crewmembers  and  mission.  It  validates  crew  currency,  and  allows  inputs  for 
a  commander  to  adjust  how  the  assessment  performs.  SARA  automates  the  general  Go-No-Go 
criteria  during  the  mission  scheduling  process  and  allows  for  electronic  input  for  tracking 
information  (i.e.,  electronic  bubble  sheets).  This  is  a  good  thing.  However,  SARA  has  inherent 
flaws.  For  starters,  it  generates  a  single  risk  number  that  is  derived  from  ordinal  scales  and  this 
number  is  used  to  trigger  a  “risk  flag”  used  to  inform  the  commander  of  a  potential  problem. 
Also,  it  allows  commanders  to  manipulate  the  assessment,  which  could  add  confusion  to  the  risk 
numbers  unless  used  properly;  this  will  require  extensive  training.  As  discussed  in  Chapter  3, 
the  use  of  ordinal  scales  presents  a  wide  range  of  problems  if  they  are  not  used  properly.  SARA 
has  the  potential  to  mislead  commanders  by  presenting  a  single  risk  number  while  masking  other 
factors  due  to  a  summing  algorithm  that  does  not  trigger  a  risk  flag  or  risk  flags  that  are  set  so 
low  they  trigger  false  signals.  These  inherent  flaws  in  SARA  need  to  he  addressed  prior  to 
the  distribution  of  the  program. 

Another  effort  to  provide  risk  information  is  with  the  joint  Risk  Management  Information 
System  (RMIS).  RMIS  is  an  Army  product  being  developed  to  provide  a  central  information 
collection  point  of  risk  information  for  major  weapons  systems."^  The  Air  Force  Safety  Center  is 
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involved  in  its  development  beeause  it  plans  to  ineorporate  a  similar  produet  for  the  Air  Force. ^ 
The  current  information  is  primarily  accident  and/or  incident  related  but  could  be  expanded  in 
the  future.  RMIS  information  is  easily  accessible  with  search  and  retrieve  functions  unlike  the 
current  Air  Force  accident  information  database.  This  is  an  excellent  product  that  will  be  value 
added  to  current  ORM  processes  once  it  is  fielded. 

RMIS  and  SARA  are  good  for  the  advancement  of  risk  information  to  each  ACC  unit.  The 
problem  is  this  information  infrastructure  is  not  available  yet.  ACC  units  are  getting  ahead  of  the 
information  infrastructure  implementation.  During  the  initial  ORM  training  at  Langley  AFB, 
part  of  the  integration  emphasized  that:  “ORM  generates  momentum  as  value  adds  are 
understood.  Avoid  getting  out  in  front  of  this  understanding.”^  The  solution  to  this  prohlem  is 
to  wait  for  the  information  infrastructure  before  attempting  to  perform  strategic  ORM. 
This  infrastructure  will  not  come  on-line  all  at  once.  Units  need  to  assess  what  information  is 
available  and  determine  when  a  strategic  ORM  process  can  effectively  be  done.  Generally 
speaking,  ACC  units  are  not  making  this  assessment  of  the  available  information  before 
attempting  strategic  ORM.  When  the  information  becomes  available,  then  ACC  units  can 
implement  strategic  ORM  functions.  In  the  meantime,  intuitive/experiential  (deliberate  and 
time-critical)  ORM  is  the  only  real  option,  which  means  ACC  units  need  to  wait  to  perform  risk 
quantification  techniques. 

The  last  category  within  the  ORM  integration  problem  was  cultural  acceptance  to  perform 
the  ORM  tasks.  There  are  several  barriers  to  getting  people  to  perform  and  there  are  no  easy 
solutions  to  these  barriers.  I  will  offer  a  few  here,  but  more  study  needs  to  be  done  in  this  area  to 
find  better  solutions.  The  main  barrier  is  getting  people  to  do  the  work.  In  The  Dan  Peterson 
Safety  Management  Series  he  states:  “If  there  is  one  problem  with  traditional  safety  programs 
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that  towers  over  all  others,  it  is  ignoring  middle  management  in  the  program.”^  Middle 
management  responsibilities  can  be  quantified,  observed,  and  measured.  Rewards  can  be 
structured  for  the  desired  behavior  and  a  system  devised  to  link  the  reward  to  the  behavior 
consistently.  The  key  to  this  middle  manager  involvement  is  a  reward  system.  Again, 
Peterson’s  research  concluded  that:  “Two  factors  determine  how  much  effort  a  person  puts  into  a 
job:  (1)  his  or  her  opinion  of  the  value  of  the  rewards  and  (2)  the  connection  the  person  sees 
between  effort  and  those  rewards.”^  Rewards  can  range  from  personal  satisfaction  of  a  job  well 
done  down  to  a  small  totem  of  recognition  such  as  an  award.  The  rewards  program  determines 
the  relationship  of  the  reward  value  and  the  effort  involved  in  the  task.  For  example,  a  small 
reward  begets  a  small  effort.  There  is  not  an  ORM  rewards  structure  in  ACC.  To  get  an 
effective  reward  system,  there  needs  to  be  an  objective  reward  criterion,  which  includes  some 
form  of  measurement.  ORM  integration  needs  some  form  of  reward  system  so  people  will  be 
willing  to  perform  ORM  tasks.  To  get  a  reward  system,  ORM  objectives  needs  to  be 
quantified,  observed  and  measured.  As  stated  in  the  previous  chapter,  observing  and 
measuring  ORM  objectives  through  a  formal  inspection  system  will  probably  fail  for  reasons 
similar  to  why  QAFA’s  were  canceled  in  the  Air  Force.  However,  setting  objectives  at  the  wing 
level  and  having  wing  leadership  devise  a  measurement  system  that  is  truly  directed  towards 
rewards  rather  than  compliance  is  a  likely  candidate.  The  ACC  Squadron  Commander’s  Course 
makes  an  attempt  to  get  new  commanders  to  induce  ORM  change.  The  course  lists  three 
inducement  examples:  (1)  show  people  how  good  it  can  be,  (2)  you  cannot  expend  unlimited 
resources  on  issues  cloaked  in  ORM,  and  (3)  commit  intelligent  application  of  ORM 
procedures.^'  The  first  of  these  could  be  part  of  a  rewards  system  in  that  people  recognize 
benefits  of  ORM  in  tangible  combat  capability.  The  second  inducement  is  the  commitment  to 
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ORM  is  zero-sum  gain  in  resources;  there  are  limited  resources  and  care  must  be  taken  to  apply 
these  resources  to  appropriate  functions.  The  third  inducement  suggests  the  use  of  proper  or 
intelligent  ORM  techniques.  In  order  to  develop  these  inducements,  ACC  units  need  more 
detailed  ORM  training  on  how  to  integrate  ORM,  and  an  information  infrastructure  so  the  ORM 
analysis  can  be  performed. 

Quantitative  versus  Qualitative  Risk  Assessments 

The  bottom  line  solution  to  quantitative  risk  management  is  do  not  do  it  until  the 
information  is  available  at  the  unit  level.  In  the  meantime,  ACC  units  should  concentrate  their 
efforts  on  deliberate  and  time-critical  ORM  techniques  using  experiential  and  intuitive 
analysis.  These  analyses  could  include  ordinal  scales  as  long  as  the  unit  is  aware  of  the 
limitations  and  they  do  not  attempt  to  sum  or  average  the  scales  to  obtain  overall  risk  levels  or 
set  levels  of  responsibility.  The  easiest  way  to  do  this  is  to  take  the  existing  risk  matrices  and 
remove  all  the  numbers  from  the  page  but  leave  the  columns.  During  the  assessment,  if  one 
or  more  elements  in  the  matrix  are  in  the  far  right  column,  then  the  person  doing  the  assessment 
uses  good  judgement  to  make  changes  or  advises  his  or  her  supervisor.  Finally,  use  these 
matrices  as  a  means  to  improve  the  existing  briefing  guides  or  use  briefing  guides  as  a 
replacement  for  the  matrices.  Briefing  guides  can  provide  as  much  information  as  the  matrices 
without  the  perception  that  the  matrix  is  a  complete  analysis  of  the  mission  risks.  Briefing 
guides  were  developed  from  experience  and  intuitive  analysis.  Current  briefing  guides  are  linear 
in  nature.  This  means  they  are  a  single  column  of  elements,  which  are  briefed  in  sequential 
order.  Matrices  can  be  used  to  create  a  two  dimensional  briefing  guide  that  include  elements  of 
severity  for  each  briefing  item.  This  new  form  of  briefing  guide  would  create  a  guide  and 
decision  aid  all  in  one.  This  is  ORM  at  its  best  in  today’s  ACC.  When  the  elements  of  the 
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briefing  guide  can  be  quantified  with  a  historical  database  that  is  when  ACC  can  move  to 
strategic  levels  of  ORM  integration. 
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Chapter  5 


Conclusions 

There  are  two  main  problems  that  exist  at  the  unit  level  in  ACC’s  implementation  of  ORM. 
The  first  is  a  general  lack  of  integration  of  ORM  into  the  mainstream  process.  The  second  is  the 
misuse  of  risk  quantification.  The  essence  of  these  problems  is  that  ACC  units  are  attempting  to 
implement  ORM  but  they  are  getting  ahead  of  the  implementation  plan.  The  plan  calls  for  basic 
ORM  training  for  all  Air  Force  personnel,  integration  of  ORM  into  mission  critical  processes 
while  developing  a  cultural  change  to  accept  ORM  principles  as  a  way  of  doing  day-to-day 
operations.  ACC  units  are  not  integrating  the  ORM  process  at  appropriate  points  in  the  mission 
process.  The  information  needed  to  integrate  ORM  is  not  available  to  ACC  units  in  a  form  that 
can  be  used  for  strategic  risk  analysis.  Most  units  are  performing  time-critical  or  deliberate 
levels  of  ORM  and  this  implies  the  use  of  intuitive  analysis.  However,  ACC  units  are  attempting 
to  quantify  this  analysis  which  is  creating  more  problems.  They  are  getting  ahead  of  the 
implementation  of  risk  information  and  required  decision  aids  to  help  them  integrate  ORM  into 
critical  mission  processes.  There  is  a  general  misuse  of  risk  quantification  because  people  do  not 
understand  how  to  quantify  risks  nor  do  they  know  the  limitations  of  risk  quantification.  Again, 
units  are  getting  ahead  of  the  implementation  because  they  don’t  understand  risk  quantification. 
Finally,  there  are  several  barriers  to  the  cultural  acceptance  for  people  to  perform  the  ORM 
functions.  People  do  not  see  enough  value  in  the  rewards  of  doing  ORM,  which  means  the  effort 
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they  put  toward  ORM  is  limited.  ACC  units  are  going  to  great  lengths  to  implement  ORM,  but 
the  problems  being  created  could  impede  the  overall  implementation  of  ORM  because  people 
will  lose  interest  in  the  program  due  to  frustration. 

The  remedies  to  these  problems  are  detailed  in  the  previous  chapter.  The  main  solutions  are 
for  ACC  units  to  be  patient  during  this  implementation  process.  The  necessary  information  to 
perform  strategic  ORM  functions  is  not  generally  available.  The  Air  Force  is  taking  steps  to 
bring  these  types  of  assets  to  the  operational  unit,  but  they  do  not  yet  exist.  The  ORM  cultural 
change  that  is  desired  will  not  occur  until  ORM  is  better  integrated  into  mission  processes  and 
the  tangible  value  of  ORM  is  realized  by  the  average  airman.  This  cultural  change  is  achieved 
through  good  training  and  effective  integration  of  ORM.  Finally,  ACC  units  should  eliminate 
the  use  of  risk  matrices  until  they  understand  risk  matrix  limitations  and  have  the  information  to 
develop  them.  Patience  is  the  key  to  ORM  implementation. 
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Appendix  A 


Examples  of  Risk  Assessment  Matrices 
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AIRCRAFT  TOWING  MATRIX 

Risk  Assessment 


To  determine  the  amount  of  risk  involved,  cirele  the  number  eorresponding  to  the  known  hazards 
listed  on  the  eharts.  Then  total  all  the  numbers  and  take  the  average  of  the  eategories  to  get  a 
final  idea  of  how  risky  the  task  will  be.  Check  the  risk  gauge  at  the  end  to  determine  how  much 
risk  is  involved.  Take  action  to  eliminate  or  reduce  risks  wherever  possible. 


A:  Are  the  personnel  experienced?  How  long  have  the  personnel  been  at  work? 


Personnel 

Hours  at  work 

Experience 

0-8  hours 

8-12  hours 

12-l-hours 

Inexperienced 

3 

4 

5 

Experienced 

2 

3 

4 

Highly  Experienced 

1 

2 

3 

B;  What  type  of  condition  is 
utilized? 

1  the  required  equipment  in?  How  often  is  the  required  equipment 

Equipment 

Equipment  utilized 

Condition 

High 

Medium 

Eow 

Poor 

3 

4 

5 

Good 

2 

3 

4 

Excellent 

1 

2 

3 

C:  Are  there  communication  devices  available  to  the  team?  How  noisy  is  the  area? 


Personnel 

Experience 

Hours  at  work 

0-8  hours 

8-12  hours 

12-l-hours 

Inexperienced 

3 

4 

5 

Experienced 

2 

3 

4 

Highly  Experienced 

1 

2 

3 

D:  What  are  the  weather  conditions  like?  What  type  of  shelter  is  available  for  task? 


Weather 

Noise 

Devices 

Eow 

Medium 

High 

None 

3 

4 

5 

Whistles 

2 

3 

4 

Radios 

1 

2 

3 

RISK  GAUGE 


POINTS 

1-5 

6-10 

11-12 

Eevel  of  responsibility 

Tow  Super 

Bear/Ram  5 

Elight/Mx  Super 

Eow  Caution 

Medium  Caution 

High  Caution 
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Example  2  -  RISK  ASSESSMENT  MATRIX 


Experience 

CDTS 

Sorties;  all 

Greater  than  500 
hours 

0 

CDTS  Sorties 
(Anyone<500 
hours)  IQC 

Sorties 
(Instructors 
>1000  hrs  or 
students  >35  hrs) 

20 

IQC  Sorties 
(Instructors  less 
than  1000  hours 
or  students  less 
than  35  hours) 

30 

Proficiency- 
Fits  in  last  60 
days;  CDTS=3 

8  or  more 

0 

4-7 

10 

Less  than  4 

20 

Crew  Rest 

Well  rested 

0 

1  crewmember 
tired 

25 

2  or  more 

crewmembers 

tired 

25 

ea 

Show  Time 

0700-1900 

0 

0545-0659  or 
1900-2100 

15 

2100-0544 

30 

Delayed  Launch 

0-2  hrs 

0 

2-4  hrs 

10 

More  than  4  hrs 

40 

Sortie  Length 

Up  to  5  hrs 

0 

5-7  hrs 

10 

More  than  7  hrs 

25 

Excessive 

Delay — Flying 
beyond  required 
activities 

Less  than  1 
hours 

0 

1-2  hours 

5 

More  than  2  hrs 

10 

Recovery  Time 
from  previous 
night  flight 

>12  hrs  land 
time — start  of 
msn  planning 
day  activities 

0 

>9  buy  <  12  hrs 
land  time-start  of 
msn  planning  day 
activities 

10 

<  9  hrs  from  land 
time  to  start  of 
mission  planning 
day  activities 

25 

Weather 

VMC 

0 

Marginal  VFR 

15 

IMC 

30 

Illumination 

Daylight 

0 

Civil  Twilight  y4 
full  moon,  non 
NVG 

15 

Less  than  14  moon 
non  NVG 

30 

Scheduled  Crew 
Duty  Day 

Day  <12  hrs 

Night  <  9  hrs 

0 

Day  12-15  hrs 
Night  9-12  hrs 

10 

Day  >15  hrs 

Night  >  12  hrs 

25 

Familiar  with 
Environment 

Very  Familiar 

0 

Somewhat 

familiar 

25 

Never  seen  before 

50 

Formation  Size 

Single  ship 

0 

2-ship 

10 

3 -ship  or  more 

25 

Mission  Planning 

No  interrupts  or 

scheduled 

activity 

0 

Some  interrupts; 
one  scheduled 
activity 

10 

Numerous 
interrupts;  more 
than  one 

scheduled  activity 

35 

Transition 

Night  %  hr  Day 

1  hr  total 

0 

Night  14-1 1/2  hrs 
total  Day  1-2  hrs 
total 

10 

Night  more  than  1 
14  hrs  total 

Day  more  than  2 
hrs  total 

20 

40 


Mission 

Complexity 

All  simulated 
bomb  runs 

0 

Live/Shape  drop 
or  FIE 

20/ 

15/ 

25 

Flag  Exercise; 
Strike  package 
with  FIE 

50 

Total 

Grand  Total 

Overall  Risk  &  Level  of  Approval  Required 


Points 

Risk 

Signature  Required 

Signature 

<50 

Very  Eow 

50-95 

Eow 

100-145 

Medium 

150-195 

Caution 

SQ  CC/DO 

200-245 

High 

OG/CC 

250  or  more 

Danger 

WG/CC  or  Higher 
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